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ABSTRACT 

Pavlov is a Programming By Demonstration (PBD) system 
that allows animated interfaces to be created without 
programming. Using a drawing editor and a clock, 
designers specify the behavior of a target interface by 
demonstrating stimuli (end-user actions or time) and the 
(time-stamped) graphical transformations that should be 
executed in response. This stimulus-response model allows 
interaction and animation to be defined in a uniform 
manner, and it allows for the demonstration of interactive 
animation, i.e., game-like behaviors in which the end-user 
(player) controls the speed and direction of object 
movement. 

KEYWORDS 

End User Programming, ULMS, Programming By 
Demonstration, Programming By Example, Animation 

INTRODUCTION 

A visitor to our planet might deduce that most computer 
users have the necessary skills to quickly and easily 
graphical user interfaces (GUIs). First, computer users 
know what they want: any user of today's popular 
applications is now quite capable of delivering a detailed 
(and passionate!) discussion on the strengths and flaws of 
computer interfaces. Second, most computer users have the 
mechanical skills required to demonstrate the appearance 
and behavior of an interface: anyone that has used a 
drawing editor knows how to draw objects with a 
computer, click on them, and transform them. 

But today's development tools have yet to fully tap die 
potential of the computer user. Though interface builders 
like Visual Basic have significanUy decreased Lie time and 
expertise necessary to build staniiard interfaces, the 
development of more graphical, animaicd interlaces is still 
mostly performed by skilled programmers. This time- 
consuming and costly development is partcularly un- 

Permission to make digital/hard copies of all or pan of ihis material for 
personal or classroom uss is granted without fee provided that the copies 
are not made or distributed for profit or commercial advantoge, the copy- 
right notice, the liile of the publication and its date appear, md notice is 
given ihat copyright is by permission of the ACM, Inc. To :opy otherwise, 
to republish, to post on servers or to redistribute to lists, requires specific 
permission and/or fee. 
CH! 96 Vancouver, BC Canada 
• 1996 ACM 0-8979 1-777-4/96/04 . S3. 50 



fortunate given the exploding demand for computer games, 
interactive entertainment, and animation in even 
"standard" interfaces, and the recognized importance of 
more end-user participation in designing interfaces. There 
has been some progress: an entire book has been 
published descrrting research systems that allow 
interfaces to be created or extended by demonstration 
rather than programming [2] ; commercially, tools like 
Macromedia's Director allow non- programmers to develop 
animation and sum; interaction. 

But none of these systems cohesively combine interactive 
techniques for specifying end-user interaction, graphical 
transformation, an 1 timing, the three primary ingredients 
of an animated interface. Director is powerful for 
specifying transformation and timing, but designing 
simple interaction requires some programming, and more 
complex interaction requires an expert. The Programming 
By Demonstration (PBD) systems in [2] present powerful 
techniques for specifying transformation and some 
interaction, but dn not provide the liming mechanisms 
necessary for animation. 

This paper presents Pavlov, a GUI development system 
based on the stimulus-response model. Stimulus-response 
provides a coheshe model for demonstrating interaction, 
transformation, an 3 liming. The model seeks to minimize 
the cognitive dissonance between concept and design 
allowing designer* to demonstrate die behavior of an 
interface exacdy as they think of it: "When I do A, B 
occurs", or "two s< conds after the start of the program, tiiis 
animation begins." Beginning with a blank target 
interface^ tabula rasa if you will, die designer uses a 
drawing too! to draw the interface, then uses the same tool 
and a clock to demonstrate stimulus-response pairs. In 
essence, the designer teaches the system in a way that is 
intuitive to humans: 

The basic ph /siological function of ihe cerebral 
hemisphere throughout the subsequent individual life 
consists in a censunt addition of numberless signaling 
conditioned stinuli to the limited number of in-born 
unconditional stimuli, in drier words, in constantly 
supplementing l ie unconditioned reflexes by conditioned 
ones. 

Ivan Pctrovich Pa v lev 
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• 1- Pavlov Development of a Driving Simulator 

^timulus-response model was first used in the 
t's'DEMO system [13] to demonstrate non-animated 
^efaaviors. The model has been extended in 
;/sd that an interface can be taught about time, 
activity, and the inherent direction of some 
These extensions allow animation as well as 
^pn to be designed within the stimulus-response 
fbrfc. It is this intersection between animation and 
jS(£on, not animation per se, that distinguishes Pavlov 
^fter PBD systems. Because of iu designers can 
jjstrate most behaviors that combine interaction and 
jjbiu including game-like behaviors in which the 
r (player) controls the speed and direction of object 
rent 

<G SIMULATOR EXAMPLE 

%4fyity simulator, the top car begins moving when 
*t>£ram begins, follows a pre-defined path around the 
~\tn stops near its starting point The bottom car 
^faying only when the driver rotates the 
l<kQj"> Its speed and direction is conirollea by the 
^ s end-user) manipulating the accelerator and the 
Wheel 

.1 shows the Pavlov environment during 
em of the driving simulator (also see the Video 
i the CHI 96 Video Program). The basic tcols are 
tfng editor in the top-right corner, the clock 
^•Jght), and the Development Mode Palette {lower- 
**" J designer uses the development modes to inform 
\ as to whether s/he is just drawing the interface 
D9de), demonstrating an end-user or time stimulus 
BjS mode), demonstrating how the system should 
a stimulus (Response or Real-Time Response 
^ testing an interface (Test mode). The designer 
p^lock to demonstrate when an operation should be 
lousing the top At: box), or if an operation siiould 
pl^j periodically (the middle Every: box). The 
^ » on the clock, labeled Record Time Stirr.ulus y 
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Figure 2. The Pavlov Editor 



allows the designer to specify that the following responses 
should be triggered by rime. 

Another important part of the environment is the editor, 
shown in Figure 2 This editor displays a textual 
description of the inurface being designed. It serves to 
provide feedback to a designer as demonstrations are 
performed, and it allows the designer to modify the 
behaviors "learned" b) the system when necessary. Details 
of this editor are provi led later in the paper. 

The first task in creating the Drawing Simulator is to draw 
the two cars, the roail, the accelerator, and the steering 
wheel. To do so, the designer selects Draw mode from the 
Development Mode P*lette. and makes use of the graphic 
primitives and grouping mechanisms available in the 
drawing palette. He jtlso provides names to the drawn 
objects for later reference. 

Next, the designer begins specifying the behavior of the 
interface. Because s/he wants the two objects shaped like 
cars to move as cars do, s/he selects each, chooses Object I 
Set Direction from the menu and enters an angle that 
defines the direction attribute of the particular car. In the 
simulator, both cars initially face straight ahead on the x- 
axis, so the designer sets the angle to 0. A vector 
emanating from its ceiter appears on each car to signify 
that the car will only move forward and backward in 
relation to its direction, and must be rotated to change 
direction. The vector coes not appear during execution of 
the target interface. 

The designer is now leady to demonstrate the stimulus- 
response behavior of toe top car. in this case, the stimulus 
is time: at time 0 (\ht beginning of execution) the car 
should begin moving. ' o demonstrate, the designer selects 
Stimulus mode, sets \h* clock At: box to 0, and clicks on 
the Record Time Slimi lus button. The system reports that 
a time stimulus has leen recorded, and automatically 
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switches to Response mode. For this behavior, the designer 
wants to demonstrate a special kind of response, a real- 
lime response, so that mode is selected. The designer then 
selects the move icon in the drawing palette and drags the 
car around the track. As s/he drags the car, it doesn f t ever 
move diagonally, but instead moves forward towards its 
nose, and rotates its base (turns) in order to folio* the 
mouse around the track. After the designer release* the 
mouse button, s/he sees from the editor that the system has 
recorded a series of discrete, time-stamped responses, 
made up of alternating MoveFbrward and Rotate 
commands. 

Next, the designer enters Test mode to visually test the 
demonstrated behavior. Immediately, the top car begins 
moving and follows the path that was demonstrated. The 
designer knows s/he could edit the recorded responses in 
the editor to modify the path, but for now s/he is satisfied 
with the behavior of the top car. 

The designer then turns his attention to the bottom car. 
The bottom car's behavior is not triggered by time, bit by 
an end-user action. Thus, instead of demonstrating a rime 
stimulus, the designer plays the role of the end-user and 
demonstrates an action. After entering Stimulus mDde, 
s/he selects the Rotate icon in the drawing palette, presses 
the left-mouse button on the rectangle denoting the 
accelerator, and rotates it clockwise some amount, say 
-0.36 radians. The system reports that a stimulus was 
recorded and automatically switches the development 
mode to Response. The system also records an implicit 
stimulus-response descriptor mapping the physical action 
used to the higher-level operation: 

(1) On Accelerator J^efiDrag -> Accelerator Jlotate 

At this point, the designer needs to demonstrate that the 
rotation of the accelerator should cause a response: of 
accelerating the movement of the bottom car. First, a/he 
enters I in the Every: box of the clock. S/he knows that 
this will cause the upcoming demonstrated response to be 
executed periodically every time frame in the target 
interface. Next, s/he moves the car some amount, sa> 17 
units. Because the car is a "directed" object, the car's 
movement is restricted: it can only be moved on the vector 
defined by its direction arrow (note that in Real-Time 
Response mode this vector is allowed to change). Ibis 
restriction is as the designer desired: in response to the 
rotation of the accelerator, s/he wants the car to move 
forward^ not change direction. The system reports the 
recorded stimulus-response descriptor containing a 
proportional constant (-47.22 = 1 11-36) 

(2) On Accelerator.Rotate(s 1 )--> 

BottomCjv.MoveForward(-47.22*sl) At 0 Every 1 
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Next, the designer cnteis Test mode to check his work. 
The top car immediately begins its path. The designer, 
playing the role of the snd-user, rotates the accelerator; / 
The bottom car begins moving, and continues to move 
even after the designer releases the mouse from the ' 
accelerator. As the car leaves the right side of the screen, it 
reappears on the left Tht designer again experiments with 
rotating the acceleratoi and notices that rotating it 1 
clockwise speeds up the car, while rotating it counter^ 1 
clockwise slows it down ; 

* -C 

1 i* 

The designer is nearly satisfied but thinks the accelerator ii 
a little sensitive. He enters the editor (see Figure 2) an<J 
selects "Accelerator" as the stimulus object. The stimuli 
"Rotate(sl)" appears in lie box labeled stimulus, and S i 
single response appears m the first cell of the score ito%j| 
labeled BottomCar. The response box below the scoi? ['i 
contains the response "MoveForward". Its parameter, as % *:g 
descriptor (2) above, is -47.22*sl. The period box contain^ 
"1". To reduce how mucl the car speeds up in response 'jjfo. 
the rotate, the designer changes the proportional faadjp^ 
from -47 to -30.0. (alternatively, the period could hav^ S 
been increased). When s1*e re-enters Test .mode, s/b^j|^f 
satisfied to see that the accelerator is indeed less sensitive!^ 

The next task is to specify the behavior of the steering 
wheel so the end- user can control the direction of tjrf^ 
bottom car. The designer enters Stimulus mode and rotit&t: [. 
the steering wheel. Then, in Response mode, s/he sets li'l^ 
Every; box in the clock to 1, and rotates the bottom c^f 
The following descriptor i 5 recorded: ^ 



(3) On Wheel.Rotate(sl)--> 

BottomCar R jtate(0.25*sl) At 0 Every 1 



To test this new behavioi, the designer once again eo 
Test mode. The top car immediately begins its path 
designer rotates the accjlerator to get the bottom 
moving, then releases tie accelerator and rotates 
steering wheel to control i s direction. He is pleased to n 



that s/he was correct io set the Every: box befbtftf. 
demonstrating the rotatkn of the bottom car: just li&;^ 
real one, the car continu 2s to turn if the steering wbe^ 
remains rotated from its oiiginal setting. " s W ; 

and sti6$v* u 



The designer continues o test the interface, 
realizes that if s/he relates the accelerator county 
clockwise past its origin, ibe car begins moving backward!* 
To alleviate this problem, s/he uses the editor to delete 
previously recorded acceierator behavior, and then rep 
demonstrates it. First, in Draw mode, s/he sets the top^left| 
point of the accelerator o i the left edge of the enclosiii$|| 
rectangle and chooses Conditions I Generate from lh|| 
menu. Then s/he demonstrates the stimulus of rotating t^ 

accelerator. A dialog apjears listing a.sel of grapbij " 

^1 




Sent by: SIEMENS TTB 

APRIL 13-18. 1996 CHI 96 



5106651331 ; 




06/18/02 9:56; JfiiE«_#333; Page 5/9 



PAPERS 



^conditions relating the stimulus object to other objects in 
lithe interface. The designer selects the condition 
ii ? Accelerator.Within(EnciosingRectangle)" f and the system 
fifecords a modified version of the originally recorded 
Stimulus-response descriptor (1). 

|§i(4) On Accelerator.LeftDrag AcceleratorRotaie 
When Accelerator. Within(EnclosingRectangle) 

^fTbe designer proceeds to demonstrate the response of 
gloving the car forward, as s/he did in the first iteration. 
^ Afterwards, s/he re-enters Test mode, and is satisfied to see 
that s/he (playing the role of the end-user) is restricted 
l^rom rotating the accelerator outside its enclosing 
^rectangle. Since the top-left point of the accelerator begins 
Ron the left edge of the enclosing rectangle, there is no way 
to rotate it counter-clockwise past its origin, so the car 
cannot move backward. 

IpTHE STIMULUS-RESPONSE MODEL 
fpThe driving simulator example illustrates many features of 
||tbe stimulus-response model, including tbe extensions that 
fallow interactive animation to be defined. This section 
Iffudescribes the model in more general terms in order lo 1) 
^explain the inferences made by the system in the example, 
and 2) bridge the gap between the specific and the general, 
|;i.e M persuade the reader that Pavlov is useful for designing 
all kinds of interfaces, not just driving simulators. 

f An interface is viewed as a stimulus-response macliine. 
Stimuli are either physical actions (e.g., drag the mouse 
|f%hile pressing the left-button), higher level operations, or 
State. The interface responds to stimuli by executing a set 
:6f time-stamped operations. Operations either create, 
^ "transform, or delete objects. The set of operations includes 
the primitives found in most drawing editors and one 
additional primitive, move forward. This additional 
v primitive allows an object to be moved while constrained 
gfato the vector defined by its direction auribute. Together. 

tbe operations offer the base functionality necessaiy to 
fj;j demonstrate nearly all animated interface behaviors. 

| The Semantics of Stimulus-Response 

J§ ; The challenge of a stimulus-response development system 
£f|8;,is to provide clear syntax and semantics for how the 
designer uses the set of physical actions and operations to 
demonstrate the behavior of the target interface. The 
JSSL'.~ syntax" of Pavlov is straight forward: the designer 
: changes development modes to inform the system whither 
his intent is to draw, demonstrate a stimulus or response, 
r.at test the interface. 



: ||£ Providing clear semantics is a more challenging problem: 
;J§g; the goal is fa tbe system to always record a stimulus- 
jy? response descriptor that perfectly matches the intent of die 




designer's demonstration (this is the primary challenge of 
all PBD systems). Pavlov uses an explanation-based 
learning approach: froci a single demonstration of a 
stimulus-response pair, de system uses domain knowledge 
and the information provided by the demonstration to 
record as reasonable a stimulus-response descriptor as 
possible. If necessary, th j designer can then use Pavlov's 
powerful editing facilities to modify the descriptor. 

This simplistic strategy differs from other systems that 
allow a designer to refiiie behavior descriptions through 
multiple demonstrations. Such an empirical-based 
learning approach allows more complex behavior to be 
specified, but complicate* the semantics of the system. 

The Semantics of a Stimulus Demonstration 

In Stimulus mode, the designer demonstrates the 
operations the end-user cin perform in the target interface. 
When die designer performs an operation in this mode, 
Pavlov records 1) a stimulus-response pair mapping the 
physical action used to the operation that was executed, 
and 2) the first part of a second stimulus-response pair that 
will eventually map die operation to one or more 
operations demonstrated .is tbe response. 

Mapping the physical ac ion to the operation is important 
because tbe drawing pahtte used during development to 
demonstrate operations does not appear when die target 
interface is executed (Te it mode). In Test mode, the end- 
user can only use the operations that the designer has 
explicidy demonstrated is stimuli, and can only access 
those operations using tie physical action (mouse button, 
auxiliary key) used in th>; demonstration. For example, in 
the driving simulator the end-user can only rotate the 
accelerator by dragging the mouse with the left-mouse 
button down, because that is how it was demonstrated. The 
designer cannot manipulate tbe car directly in any manner, 
because no such stimulus was demonstrated. This positive 
example method of specifying the functionality of the 
system is in contrast to the scheme of [9] in which the 
designer "freezes" the ob ects that cannot be manipulated. 

The second recording made from a Stimulus 
demonstration records the high-level operation 
demonstrated as a stimulus (e.g., AcceleratorJRotate). It is 
the execution of this high-level stimulus that will trigger 
the execution ot the resjionses demonstrated in Response 
mode. 

The only complication to die semantics of a stimulus 
demonstration is that i designer may demonstrate a 
stimulus on a representative object At run-time, the same 
stimulus applied in any nember of the set represented will 
trigger the demonstrated response. Dynamically allocated 
objects, which can be s;)ecified by creating an object in 
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stimulus or response mode (see [13]), are by default 
marked as representative objects. Pavlov also allocs the 
designer to designate behavior groups, and marks each 
element as representative of the group (a similar approach 
is used in f 12)). 

The Semantics of a Response Demonstration 
When the designer demonstrates an operation R on object 
O in response mode, the system connects a response of the 
fonn X>.R (r }t T U J when C to the previously 
demonstrated stimulus, where each r, is a function of zero 
or more stimulus parameters, and C is an optional context 
for when R should be executed. 

The simplest semantic rule is to execute the demonstrated 
response each time the demonstrated stimulus occurs :n the 
target interface. However, such a simple rule would 
preclude the designer from demonstrating the context for 
when an operation should be executed; semantics su:h as 
"execute R is response to stimulus S only wher. the 
environment is in state s" could not be demonstrated. 

By setting a toggle, the Pavlov designer explicitly states if 
context should be taken into account. If it is, the designer 
configures the interface into the desired context (or the 
negation of the desired context) prior to a response 
demonstration. After the demonstration, the system nins a 
set of tests to identify graphical conditions describing the 
state of the interface. The designer is allowed to select one 
or more of these conditions and combine them with logical 
operators to define the context for when a response should 
be executed. 

In the driving simulator, a context was defined on the 
description mapping the physical stimulus 
^Acceimtor.LeftDrag" and the response 
"Accelerator.Rotate". A context could also be demonstrated 
so that the bottom car doesn't run into the top one: the 
designer demonstrates the stimulus of rotating the 
accelerator, tells the system to identify context, then 
demonstrates a response of moving the bottom car so that 
it intersects the top car. When the system identifies 
BottomCar.Intersects(TopCar) amongst other conditions, 
the designer selects it and negates it, and the following 
behavior is recorded: 

(5) On Accelerator J*oiate(sl)-> 

BottomCar.MoveForward(47.22*sl) At 0 Every I 
When Not (BottomCar.IntersectsCTopCar)) 

la general, there are many true conditions concerning the 
state of the interface. To reduce the number of conditons 
listed for the designer, the system only identifies those 
conditions relating the response object and all other objects 
in the interface. Because the stimulus object has also teen 
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signified as important, relationships found concerning ^ 
and the response object are shown at the top of the list*of ^ 
found conditions. When necessary, the designer can use?f 
the editor to specify i. condition not identified by the 
system. ' ^ 

A second complication io the response semantics is simitefi 
to that discussed in the stimulus section: if 
demonstrated response object is representative of a set, the^ 
response is applied to the entire set during execution (or {jap 
subset, as defined by a context conditional [13]). 

A third complication tc the response semantics concent 
determining the response parameters. The simplest 
solution is, of course, tc execute R during execution witfc 
the same parameters as in the demonstration. This is tjti 
best solution when the corresponding stimulus has ii| 
parameters (e.g., the st mulus is a button click and thi|| 
response is a move(x=5,j^7)). However, for stimuli thatj^ 
have parameters, it is often the case that the reactioifeff 
proportional, i.e., the response parameters) aftp 
proportional to the p;irameters of the stimulus, fift 
instance, the car in the Driving Simulator is rotated^ 
amount proportional to the amount the steering wheel if 
rotated. Thus, when such a stimulus-response \M 
demonstrated, Pavlov irfers proportional constants 
r t /Si. that relate each of the stimulus and response 
parameters. When the stimulus S(sj,s 2 ,...) occurs 
execution, the response K (s^C,. s 2 . C 2 ) is executed. 

The formula for R illustrates that the system infens the 
parameter of the stimulus to be related to the 
parameter of the response :, the second to the second, an 

on. The basis of this nference is that most hue. 

operations either have a :iingle parameter or they have i^ci 
parameters denoting x and y coordinates, so in practice' 
corresponding stimulus and response are often related;: 
with conditionals, the alitor can be used to modify? 
response formulas record xi ^ 

EXTENSIONS FOR ANIMATION 

Systems for demonstrating animation have existed for 
twenty-five years [1J. Pavlov's contribution is 
integration of animation demonstration with the 
response model for defining interaction. 

An animation path can le demonstrated with a real 
response demonstration, with a series erf time-stam 
response demonstrations (the editor can be used for: 
bctweening), or with a periodic response. Like any ot 
response, an animation p uh can be triggered by any 
of end-user or tune stimulus. 

When a designer demonstrates the transformation of 
object iii real-time response mode, die system reconcj 
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i pf lime-stamped operations. Because operations are 
instead of picture frames (as in Director), the 
pded path is not constrained to a particular starting 
jji or object Thus, reuse is facilitated. The mech:inism 
po slightly more general than in systems such as 
pr because any operation, not just move, am be 
strated in real-time. 

Option of Direction 

jjppportant contribution of Pavlov is that a designer can 
ajpnstrate animation in which the end-user not only 
*#e$ movement, but accelerates it and changes its 
jon. Though such behavior is the primary activity in 
^game-like applications, there has been little research 
area, and commercial systems such as Director 
extensive programming to develop this part of an 
Ication. 

_^-J struggling with bow to allow game-like behav:or to 
ijifpQnstrated, the following observations were made: In 
jp games, one input control (e.g., steering whe;l) is 
" to control direction, and a different control 
Aerator) to control speed. Also, many objects do not 
j in an arbitrary manner, but are restricted to moving 
vard and backward, and must rotate their base to turn, 
j these observations it became clear that the standard 
ve(x,y) operation in Pavlov's drawing editor is. not 
"cient for the demonstration of movement because it 
ifies both a distance and an absolute direction. 

||plve the problem, a notion of direction was addid to 
\stjmulus-rcsponse model. Designers can set a current 
Hon attribute for an object that is displayed during 
elopment The direction attribute makes it possible for 
[designer to demonstrate a MoveForward(d) operation, 
operation causes an object to move forwarc: (or 
if d<0) in the direction it is facing. Thus, it is 
cb better suited for the demonstration of acceleration 
iMove(x,y). 

jtrtodk Responses 

notion of a periodic response is also useful in 
£monstrating acceleration. In the driving simulator, when 
1^ end-user rotates the accelerator, the car should tegin 
paving and continue to move, even after the end user 
peases the mouse. In Pavlov, the designer explicidy 
Res continuous movement by setting the Every; box 
pp. the dock before the demonstration of a forward 
Pavement In essence, when a "MoveForward (d) Every t" 
^demonstrated, the system infers that the object should 
; forward at a speed of d/t 

@®e following run-time rule follows from these semantics: 
execution of successive periodic MoveForward 
Iterations on die same object results not in two alternating 
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and possibly opposite actions, but in a single action 
combining the magnituces of the operations. For example, 
in the driving simulator, when the car is already moving 
at 4 units/frame and the end-user rotates the accelerator 
again, say back towards the origin, it causes a response of 
MoveForward (-1) unite /frame. This second operation is 
combined with the existing one so that the car slows down 
to 4-1 =3 units/frame, instead of alternating between 
moving forward 4 units, and backward 1 unit. 

The system only uses ttese semantics for periodic Move- 
Forward operations. For other operations, successive 
periodic responses will eiecute in tandem. Thus, using two 
periodic, regular Move demonstrations, the designer can 
demonstrate that an object move back and forth, such as in 
an animated move icon. 

An alternative method of demonstrating acceleration has 
also been added to toe Pavlov environment After 
demonstrating a stiff ulus that should cause the 
acceleration, the design* r enters real-time response mode 
and moves an object forward at the desired speed. 
Generally, a real-time response is used to demonstrate a 
fixed animation path as a response to a simple button-click 
or time. When a real-time Move Forward is demonstrated 
as a response to a transformational stimulus (one with 
parameters), the system loes uot record a series of discrete 
time-stamped operations as usual, but instead records a 
single periodic operation. The distance parameter is 
computed by dividing; the total distance of the 
demonstrated movement by the time of the movement (d/t), 
and the period is set to 1 (ms). 

The advantage of this scheme is that the designer truly 
demonstrates the speed of the movement; the disadvantage 
is it complicates the semantics of the system. A more 
thorough analysis will bu provided after more feedback is 
gathered from users. 

THE PAVLOV EDITOR 

An important aspect of i PBD system is how a designer 
edits the behaviors inferred by the system. Pavlov's editor 
borrows from Director ty providing a time-line view of 
activity (a score). However, because interaction is 
emphasized, Pavlov pro\ ides multiple timelines: one for 
the events that occur wthout an end-user stimulus, and 
one fear the events triggered by each end-user stimulus that 
was demonstrated. This method of organizing events by 
stimulus significantly ea;es the editing task compared to 
the single score editors fcund in most animation systems. 

Pavlov's editor, shown in Figure 2, can be viewed 
simultaneously with the main development window. In 
order to view the operatons that occur in response to a 
particular stimulus, the iesigner selects an object and a 
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particular stimulus in the top-left list boxes. To view the 
operations that occur without an end-user stimulus, the 
designer selects the "Time Stimulus" check box to the 
right of the stimulus list 

The objects that respond to the listed stimulus are sboun in 
the rows of the score. The designer can select a particular 
response in a cell, and the inferred response parameters 
appear in the edit boxes labeled Rl and R2. Any 
expression consisting of constants, stimuli parameters, and 
system-supplied object attribute functions may be entered 
as a response parameter. In essence, editing behavior 
formulas is very similar to entering a formula in a 
spreadsheet 

IMPLEMENTATION 

At the beginning of execution (Test mode), the time 
stimulated operations defined in the target interface are 
placed in the execution list with their respective time 
stamps. For each end-user stimulus that occurs, the sr 
processor traverses the selected object's stimulus-response 
list to find the response operations associated with the 
stimulus. These responses are copied into the execuUon list 
with a time-stamp of t + t, , where t is the time the 
stimulus occurred (the current time), and t, is the reconled 
time-stamp of the response (which is relative to the 
stimulus). When the system is not processing end-user 
stimuli in this manner, the execution list is traversed and 
all operations whose time-stamp is less than the current 
time are executed. A non-periodic operation is removed 
from the execution list immediately after execution; a 
periodic operation is left in the list, with its time-stamp 
incremented by the size of its periodic interval. In either 
case, the executed operation is sent as a stimulus to the sr 
processor, so a chaining of events can occur. 

Though the scheme does not guarantee that operations w ill 
be executed before or on their time-stamp, in practice it 
provides visually acceptable performance even for 
interfaces with lots of interaction and concurrent 
animation (Pavlov runs on a 486 PC). 

RELATED RESEARCH 

Rehearsal World [S] and Peridot [7] were early PBD 
systems that inspired the stimulus-response framework. 
The first systems to allow direct graphical demonstration 
of a full range of stimuli and responses were DEMO [13] 
and Marquise [8]. DEMO introduced the stimulus- 
response model and a technique for demonstrating 
dynamically created objects, while Marquise focused on 
the demonstration of graphical editors, including those 
with palettes and modes, 

O£M0 II [3J and [4] are stimulus-response systems that 
allow the designer to perform multiple demonstrations of 
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the same behavior to refine the system's inferences |*u 
uses multiple examples to make sophisticated mferenS 
concerning response parameters- inferred formulas 3l 
depend on attributes :>f arbitrary objects as well^t 
stimulus parameter vaJues. DEMO II uses multi^ 
examples to refine inferences concerning the context S 
when a response should b i executed. ■ 

Pavlov is the first stimi lus-re sponse system to focus^a 
animation, though there ;ire a few PBD systems not ha^ 
on stimulus-response that allow some animation to** 
demonstrated: KidSim \\2) and Agent Sheets [10] 
graphical rewrite rules tc allow designers to demonsti 
the context for when an operation should be executed, 
systems are powerfa for creating non-intera4 
simulations, but the rewrite-rule method of defihL^ 
context is not integrated *ith a method of specifying eBp 
user stimuli, so interactive simulations cannot be desigft * 
without coding; Dance [1] allows the demonstrate 
animation to the purpDse of program visualizatif 
Chimera [6] and LEMMIVG [9] allow interface behav 
to be specified with multiple demonstrations^ 
constraints, but do not :ovcr time-based animation^ 
acceleration. 

Director is representative of the commercial animal! 
systems that provide facilities for both animation & 
interaction design. These systems allow animation ttfto 
designed quickly and casil/ using a combination of (r t 
by-frame animation, in-betweening, and realty 
recording. These systems *lso allow sound and video toil 
linked into presentations, md provide a range of featuifc 
for creating special effects such as sbw-in/sbwi& 
motion blur, and squash a id stretch. -nlgf 

Though powerful for defin ng animation, these systems i 
not provide a PBD methed of defining interaction/ y 
systems all allow button-click triggered animation to", 
defined in a relatively sinple manner However, rittL, 
complex stimulus-response behaviors, such as the steerinjj 
wheel and accelerator controlled animation in the drivtf 
simulator, require expert-le /el programming. 

A second difficulty in defiling interaction with animatie 
systems is that they are based on a single-score editor! 
the animation sequences of an application are shown difip 
single time-line. Though su:h a score is sufficient for L I 
interactive animation (whici was its original purpose);^ 
too unstructured for applica ions with interactive as weli V 
time-stimulated animation. Like the programs wrii 
before the advent of structured programming (sill 
procedures), the designer is forced to program control, % 
where one animation ends und another begins, using - 
statements. For complex applications with lotS; 
movement and interaction, the result is a spaghetti sc 
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Multiple score scheme in the Pavlov editor alleviates 
oblem, and allows the designer to edit the different 
live behaviors and animation sequences separately. 

tATIONS 

tnajor practical limitation of Pavlov is that interfaces 
with it cannot be connected to application code. In 
njjext version, designers will be able to make this 
ion by 1) demonstrating a function call iis a 
utus or response, and 2) calling an application 
on within a response or conditional formula, 
ji^'s single demonstration scheme might also be 
dered a limitation: more behaviors could be inferred 
Jftultiple demonstration inference engine, such as [4], 
iegrated. Before doing so, however, we want to study 
liber the additional inferred operations justify the 
ibnal complexity that would be added to the 
nment. A third limitation is that acceleration csn be 
^strated for an object that has no pre-defined path, 
Cannot be demonstrated for an object that must stay on 
d path. In this regard, we are exploring both the use 
necric functions to model some animation paths, 
the use of "conductor" objects with special properties 
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"V contributes a cohesive model for demonstrating 
Btion and interaction, and innovative techniques for 
^nstrating interfaces in which the end-user controls 
the speed and direction of animation paths. Using 
techniques, interfaces like the driving simulator can 
created in less than fifteen minutes. 

elopment is by no means restricted to driving 
viators or similar applications. A number of other 
faces have also been developed, including a wide 
Sty of games, a diagram editor with animated icons, 
an educational solar system program. We attribute 
general usefulness of Pavlov to the generality of the 
jiilus-response model, and its powerful multiple-score- 
I editing facility. 

les increasing the range of PBD, the stiirulus- 
ppnse model provides a very intuitive method for 
^fining interfaces. A usability lest was performed with a 
fpiber of non- technical designers. Subjects were given a 
ual describing the stimulus-response model, and then 
e asked to design two interfaces equal in complexity to 
driving simulator. Seven of ten were able to create the 
ferfaces within an hour, the three others completed the 
after asking a few questions (specific questions 
;rning how to complete the tasks were not ailcwed). 
were extremely encouraged by the results, as well as 
the enthusiasm the subjects expressed for exploring once 
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the formal tests were completed (though we could get 
none to salivate!). 
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Pygmalion was an early attempt to change the process of programming from one 
in which algorithms are described abstractly in a programming language to one 
in which they are demonstrated concretely to the machine. It introduced two new 
concepts: programming by demonstration and icons. Icons have become widely 
accepted. Programming by demonstration is still waiting for a practical system 
to show its validity. Pygmalion was completed in 1975. This chapter is a com- 
mentary on that work. 



Traditionally people instruct computers by sitting down with a pad of paper (or a 
window for a text editor) and writing a sequence of statements in some pro- 
gramming language. They then compile the statements, link the resulting ma- 
chine code with other run-time routines, and attempt to execute it There are al- 
most always errors. At this point programmers enter the debug-edit-recompile 
loop. They track down bugs using whatever means are available, edit the source 
text of their programs, recompile it, relink it, and reexecute it. There are almost 
always more errors. In fact, fixing a bug often introduces new bugs. This debug- 



edit-recompile loop can consume the majority of time sprat on a program. And 
no one claims anymore that all bugs can be removed from any large program. 

Writing static language statements interspersed with compile-run-debug-edit 
periods is obviously a poor way to communicate. Suppose two humans tried to 
interact this way! Specifically, it is poor because it is: 

(a) Abstract 

The programmer must mentally construct a model of the state of the machine 
when the program will execute, and then write statements dealing with that 
imagined state. In practice, for a program of any complexity this is impossible. 
People cannot handle as much complexity as computers can. A typical symptom 
of this is boundary errors: an unanticipated value is used in an operation, result- 
ing in an error. Dividing by zero is a classic example. We must find a way to al- 
low programmers to work with concrete values without sacrificing generality. 

(b) Non-interactive 

The debug-edit-recompile loop takes a lot of time per bug fix, particularly as 
programs get large. This low productivity has led programmers to seek faster 
machines and compilers. But that misses the point Instead of speeding up the 
debug-edit-recompile loop, programmers should eliminate it! A few program- 
ming languages such as Lisp and Smalltalk are in fact interactive. One can type 
in an expression and immediately get it evaluated and see the result Such lan - 
guages are more productive than non-interactive languages. 

So why aren't interactive languages more widely used? Most are high level lan- 
guages that are not as efficient as low level ones such as C and FORTRAN. In 
their search of a competitive advantage, software developers want their pro- 
grams to execute as fast as possible, even at the expense of a longer develop- 
ment time. But this situation may be changing. Computers are now fast enough 
that interactive languages are practical in some cases. Several commercial appli- 
cations have been written in Lisp, and IBM now recommends Smalltalk for de- 
veloping applications under its OS/2 operating system. Even some versions erf C 
now contain an interpreter for quick development and debugging. 

(c) "Fregean" 

The most articulate representation for a program requires the least translation be- 
tween the internal representation in the mind and the external representation in 
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the medium. Aaron Sloman [Sloman 71] distinguishes two kinds of data repre- 
sentations: "analogical" and "Fregean" (after Gottlob Frege, the inventor of the 
predicate calculus). Analogical representations are similar in structure to the 
things they describe; Fregean representations have no such similarity. Consider 
the following examples: 

1.23 
45.98 

17.1 
63.709 
125.4 



triangle 



United 
States 



bicycle 

Analogical Fregean 

One of the advantages of analogical representations over Fregean ones is that 
structures and actions in a context using analogical representations (a 
"metaphorical" context) have a functional similarity to structures and actions in 



array A 






the context being modeled. One can discover relationships in a metaphorical 
context that are valid in the other context For example, there are five items in 
array A; triangles have three sides; bicycles have two wheels; Texas is in the 
south. Analogical representations suggest operations to try, and it is likely that 
operations applied to analogical representations would be legal in the other con- 
text, and vice versa. This is the philosophical basis for the design of the Xerox 
Star and, ultimately, Apple Macintosh "desktop" user interfaces; they are a 
metaphor for the physical office [Smith 82]. Being able to put documents in 
folders in a physical office suggests that one ought to be able to put document 
icons in folder icons in the computer "desktop," and in fact one can. 

Jerome Bruner, in his pioneering work on education [Bruner 66], identified three 
ways of thinking, or "mentalities": 

• enactive - in which learning is accomplished by doing. A baby learns what a 
rattle is by shaking it. A child learns to ride a bicycle by riding one. 

• iconic - in which learning and thinking utilize pictures. A child learns what 
a horse is by seeing one or a picture of one . 

• symbolic - in which learning and thinking are by means of Fregean symbols. 
One of the main goals of education today is to teach peqple to think 
symbolically. 

But Bruno- argues that it is a mistake for schools to force children to abandon 
their enactive and iconic thinking dolls when learning the symbolic. All three 
mentalities are valuable at different times and can often be combined to solve 
problems. All three skills should be preserved. 

Jacques Hadamard, in a survey of mathematicians [Hadamard 45], found that 
many mathematicians and physicists think visually and reduce their thoughts to 
words only reluctantly to communicate with other people. Albert Einstein, for 
example, said that "The words of the language, as they are written or spoken, do 
not seem to play any role in my mechanism of thought The psychical entities 
which seem to serve as elements in thought are certain signs and more or less 
clear images which can be 'voluntarily' reproduced and combined.... This com- 
binatory play seems to be the essential feature in productive thought— before 
there is any connection with logical construction in words or other kinds of signs 
which can be communicated to others.... Hie above mentioned elements are, in 
my case, of visual and some of muscular type. Conventional words or other 
signs have to be sought for laboriously only in a secondary stage, when the 
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mentioned associative play is sufficiently established and can be reproduced at 
will." [Hadamard 45, pp. 142-3] 

Some things are difficult to represent analogically, such as "yesterday" or "a 
variable-length array.'* But for concepts for which they are appropriate, analogi- 
cal representations provide an intuitively natural paradigm for problem solving, 
especially for children, computer novices, and other ordinary people. And a 
convincing body of literature suggests that analogical representations, especially 
visual images, are a productive medium for creative thought [Arnheim 71], 

Words are Fregean. Yet words and rather primitive data structures are often the 
only tools available to programmers for solving problems on computers. This 
leads to a translation gap between the programmer's mental model of a subject 
and what the computer can accept / believe that misunderstanding the value of 
analogical representations is the reason that almost all so-called visual pro- 
gramming languages, such as Prograph, fail to provide an improvement in ex- 
pressivity over linear languages. Even in these visual languages, the representa- 
tion of data is Fregean. 

With the advent of object-oriented programming, the data structures available 
have become semantically richer. Yet most objects are still Fregean. A pro- 
grammer must still figure out bow to map. say, a fish into instance variables and 
methods, which have nothing structurally to do with fish. 

(d) "Blank canvas" syndrome 

Pablo Picasso said "the most awful thing for a painter is the white canvas." One 
of bis most memorable paintings, "Hie Studio at Cannes," is of his own studio. 
Its walls and ceiling are covered with paintings. Priceless works of art are jum- 
bled together everywhere. But right in the middle of the room stands an easel 
holding a blank canvas. This is a fitting image of the problem facing all creative 
people: how to get started. A blank coding pad is as much a hairier to program- 
ming creativity as a blank canvas is to a painter. 

What's needed is a lightweight, non-threatening medium like the back of a nap- 
kin, wherein one can sketch and play with ideas. Hie computer program de- 
scribed in the remainder of this chapter, Pygmalion, was an attempt to provide 
such a medium. Pygmalion was an attempt to allow people to use their enactive 
and iconic mentalities along with the symbolic in solving problems. 
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In Roman mythology, Pygmalion was a sculptor of extraordinary skill. At the 



peak of bis powers, be determined to create a masterwork, a statue of a woman 
so perfect that it would live. The statue, which he named Galatea, was indeed 
beautiful, but alas it remained just stone. Heartbroken, he prayed to the gods. 
Venus, impressed with his work and passion, took pity on him and brought the 
statue to life. (They could do that back then.) 

This urge to create something living is common among artists. Michelangelo is 
said to have struck with his mallet the knee of perhaps the most beautiful statue 
ever made, the Pieta, when it would not speak to hint And then there' s the story 
of Frankenstein. Artists have consistently reported an exhilaration during the act 
of creation, followed by depression when the work is completed. "Tor it is then 
that the painter realizes that it is only a picture he is painting. Until then he had 
almost dared to hope that the picture might spring to life." (Lucien Freud, in 
[Gombrich 60], p.94) This is also the lure of programming, except that unlike 
other forms of art, computer p ro gram s do "come to life" in a sense. 

The Pygmalion described in this chapter is a computer program that was de- 
signed to stimulate creative thinking in people [Smith 75]. Its design was based 
on the observation that for some people blackboards (or whiteboards these days) 
provide significant aid to communication. If you put two scientists together in a 
roan, there had better be a blackboard in it or they will have trouble communi- 
cating. If there is one, they will immediately go to it and begin sketching ideas. 
Their sketches often contribute as much to the conversation as their words and 
gestures. Why can't people communicate with computers in the same way? 

Pygmalion is a two-dimensional, visual programming environment implemented 
on an interactive computer with graphics display. (Although this work was com- 
pleted nearly two decades ago, I will describe it in the present tense for readabil- 
ity.) It is both a programming language and a medium for experimenting with 
ideas. Communication between human and computer is by means of visual enti- 
ties called "icons/* subsuming the notions of variable, data structure, function 
and picture. Icons are sketched on the display screen. The heart of the system is 
an interactive "remembering" editor for icons, which both executes operations 
and records them for later reexecution. The display screen is viewed as a picture 
to be edited. Programming consists of creating a sequence of display images, the 
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last of which contains the desired information. Display images are modified by 
graphical editing operations. 

In the Pygmalion approach, a programmer sees and thinks about a program as a 
series of screen images or snapshots, like the frames of a movie. One starts with 
an image representing the initial state and transforms each image into the next 
by editing it to produce a new image. Hie programmer continues to edit until the 
desired picture appears. When one watches a program execute, it is similar to 
watching a movie. The difference is that, depending on the inputs, Pygmalion 
movies may change every time they are played One feature that I didn't imple- 
ment but wish I had is the ability to play movies both backward and forward; 
then one could step a program backward from a bug to find where it went 
wrong. 

Thae are two key characteristics of the Pygmalion approach to programming. 

• It relies on editing an artifact rather than typing statements in a program- 
ming language. Editing has proven to be easy for people. Everyone who 
uses computers— over 100 million people today— can use text and graphics 
editors, but hardly anyone can "program." 

• The screen images always contain concrete examples of the program's data. 
This eliminates an entire class of errors due to abstraction. 

Today we recognize these as being good user interface principles that most per- 
sonal computer applications heed. So one way to characterize Pygmalion is to 
say that it attempted to bring good user interface principles to the process of 
programming, not just to the result of programming. 

Another early attempt to specify programming by editing was Gael Curry's 
Programming by Abstract Demonstration [Curry 78]. It was similar to 
Pygmalion, excq>t that instead of dealing with concrete values, Curry manipu- 
lated abstract symbolic representations (e.g. "n") for data. While this made some 
things easier to program than in Pygmalion, it is interesting to note that in his 
thesis he reported making exactly the kinds of boundary errors mentioned above 
because he could not see actual values. 





We will illustrate the Pygmalion way of programming by defining the function 
"factorial." An ALGOL-like definition might be: 



integer function factorial (integer n) 
if n = 1 then 1 
else n * factorial (n - 1); 



In Pygmalion, a person would define factorial by picking some number, say "6", 
and then working out the answer for factorial(6) using the display screen as a 
"blackboard." In the illustrations below, in order to save space I won't show the 
entire display screen for each "movie frame." Rather, I'll often "zoom in" on the 
parts of the screen that change from frame to frame. The programmer, however, 
always sees the entire screen: 



(Aside: the screen soots in this chapter are from a recent HyperCard simulation 
of Pygmalion done on a Macintosh computer by Allen Cypher and myself 
Therefore * e window appearance differs slightly from the actual system.) 
The PygmaUon window consists of several parts: 

• "menu" area - a menu of icons and operations on icons (a rather mixed bag 
actually-today the icons would be in a palette and the operations in pull- 
down menus). The icons are under the "opcodes" and "control" headings 
and the operations are under the "icons" and "others" headings. This is the 
complete icon editor. 

• "mouse value" area - Programmers could type a value here, and it would 
become "attached" to the mouse. It could then be deposited in icons 

• "remembered" area - When the system is "remembering," the last couple of 
operations performed are displayed here. 

. "Smalltalk" area - Smalltalk expressions could be typed and evaluated here 

• "mouse" area - Pygmalion was implemented on a computer having a three 
button mouse; this area displayed the meaning of each button in the differ- 
eat modes. 

Except for the menu, these parts are not really important As I mention at the 
end, I would get rid of them all were I to reimplement the system today But 
with these as our resources, let's define factorial 

The entities with which one programs in Pygmalion are what I call "icons " An 
icon is a graphic entity thai has meaning both as a visual image and as a machine 
object Icons control the execution of computer programs, because they have 
code and data associated with them, as well as their images on the screen. This 
distinguishes icons from, say. lines and rectangles in a drawing program, which 
have no such semantics. Pygmalion is the origin of the concept of icons as it 
now appears in graphical user interfaces on personal computers. After complet- 
ing my thesis, I joined Xerox's "Star" computer project The first thing I did was 
recast the programmer-oriented icons of Pygmalion into office-oriented ones 
representing documents, folders, file cabinets, mail boxes, telephones, wastebas- 
kets. etc. [Smith 82J. These icons have both data (e.g. document icons contain 
text) and behavior (e.g. when a document icon is dropped on a folder icon, the 
folder stores the document in the file system). This idea has subsequently been 
adopted by the entire personal computer and workstation industry 
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To get started, I define an icon for the function: 



I create two sub-icons to bold the argument and value, and then draw some 
(crude) graphics to indicate the icon's purpose: 



9-n 



Finally I make the outside border invisible: 

□o-n 

i 1 o I 1 

I now select the icon, invoke the "define" operation, and type in the name 
"factorial." Ibis associates a name with the icon. The first thing the system does 
with a newly created function is to capture the screen state. Whenever the func- 
tion is invoked, it restores the screen to this state. This is so that the function will 
execute the same way regardless of what is on the screen when it is called. So 
before invoking "define" I make sure that I clear off any extraneous icons lying 
around. Icons may be deliberately left on the screen when a function is defined; 
these act as global variables. When the function is invoked, the "globaT icons 
and their current values are restored to the screen (if they weren't already pre- 
sent) and are therefore available to the function. 

As soon as I invoke die "define" operation, the system enters "remember mode," 
causing every action I subsequendy do to be recorded in a script attached to the 
icon. So one defines functions simply by working out examples on the screen. 
Programs are created as a side effect Here I'll arbitrarily work out the value of 
factorial for the number "6". (Choosing good examples is an art One wants both 
typical values and boundary cases. Even in conventional programming, choosing 



test cases poorly is a principal sources of bugs. Pygmalion does not solve this 
problem.) 

I type "6" into the argument icon: 

o-r 



6 



o 



Whenever all the arguments to a function are filled in, the system immediately 
invokes it This was an experiment and is not inherent in programming by 
demonstration, or even necessarily a good idea. Functions in Pygmalion can be 
invoked even if their code is undefined or incomplete. When the system teaches 
a part of the function that has not yet been defined, it traps to the user asking 
what to do next This was implemented by placing a "trap" operator at the end of 
every code script When a trap operator is executed, the system reenters "record 
mode." Every action subsequently performed is inserted in the code script in 
front of the trap. When the user invokes the "done" operation indicating that this 
code script is finished, the system leaves "record mode" and removes the trap 
operator. Otherwise the trap operator remains there. 

Factorial is now ready to execute, so the system invokes it Since there is no 
code defined for it yet. it immediately traps to the programmer asking what to 
do. I move the factorial icon out of the way to make some room on the screen- 
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Pygmalion demo 
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From heie on, in order to save space in these pictures I won't show the window 
title or any area that is not directly involved in the actions being described. 

I want to test if the argument is equal to 1. Actually, I can see that it isn't 1; it's 
6. So why do I have to make a test? The answer is that Pygmalion is designed 
for programmers. Programmers know that functions will be called with different 
arguments, and they try to anticipate and handle those arguments appropriately. 
This is a little schizophrenic of Pygmalion: on the one hand it attempts to make 
programming accessible to a wider class of users; on the other, it relies on the 
kind of planning which only experienced programmers are good at But anticipa- 
tion is not inherent in programming by demonstration. Henry Liebennan's 
Tinker [Liebennan 80, 81, 82] doesn't force users to anticipate future arguments. 
In Tinker one writes code dealing only with the case at hand. When Tinker de- 
tects an argument that is not currently handled, it asks the programmer for a 
predicate to distinguish the new case from previous ones. The programmer then 
writes code to handle the new case. Tinker synthesizes a conditional expression 
from the predicate, the new code, and the existing code. Programmers never 
have to anticipate; they need only react to the current situation. This is an im- 
provement over Pygmalion 's approach. 
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At any rate, I've decided that I will need a conditional, so I invoke the T item 
in the menu. The system enters a mode waiting for me to specify where to put 
the conditional icon. When I click the mouse, the icon is placed at the cursor lo- 
cation: 



menu 



icons 
create 
change 
delete 
copy 
refresh 
shov 
name 
value 
shape 
body 

opcodes 
* 



0 = 



? 




true 


X — » 



The conditional icon consists of three sub-icons: one for the predicate and two to 
hold the code for the true and false branches. The predicate icon is the 
"argument" to the conditional; as soon as it has a value, the appropriate branch 
icon executes. 



To test if the factorial argument is equal to 1,1 dick on the menu item, point 
where I want the icon to go, and the system creates an equality-testing icon: 



It has two sub-icons to hold the data being tested, I drag the **6" down from fac- 
torial's argument icon and deposit it in the left hand sub-icon: 





CO 









(During playback— "watching the movie,"— Curry's system animated such 
value dragging, making for a quite articulate movie; this was an improvement 
over Pygmalion, which did no such animation.) Thai I type a u r into the right 
hand sub-icon: 





6 




1 





The equality icon is now fully instantiated, so it executes. It is defined to replace 
its contents with the result of the test Since 6 is not equal to 1, it replaces its 
contents with the symbol "false": 



false 
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This value, like the value in any icon on the screen, can be used in other parts of 
the computation. In particular, it can be used in the conditional icon. I drag it 
into the conditional's predicate icon. The conditional icon is now fully instanti- 
ated, so it executes. The symbol "false" causes the false icon to be evaluated: 
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But the false icon has no code defined for it yet, so it immediately traps back to 
me asking what to do. The false icon now becomes the "remembering" icon; 
previously it was the icon for factorial itself. So now I'm no longer defining 
code for the factorial icon; I'm defining code for the false branch of its condi- 
tional icon. In fact, from here on out all of the operations I perform will be at- 
tached to either the true or false branch of the conditional. 

Since I don't need the equality testing icon anymore, I'll get rid of it. I do this by 
invoking the "delete" menu item and then clicking on the equality icon. 
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Now I want to compute n* factor ial ( n- 1 ) . I* II need a multiplication icon 
and a subtraction icon. These I get by invoking the appropriate menu items: 
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I drag factorial's argument (6) into the left hand icon in each of these. Then I 
type a 1 into the right hand icon in the subtracter 
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The subtraction icon is now fully instantiated, so it executes: 



5 



Now I want to call factorial recursively on this value (5). I invoke the "call" 
menu item, type "factorial" when prompted, and dick whore I want it to go. Hie 
system creates a new instance of the factorial icon, the very one that I am in the 
middle of defining! As mentioned earlier, that factorial is not completely defined 
poses no problems for Pygmalion; it will execute what it has and ask for more 
when it runs out. The screen now looks like this: 
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I drag the 5 into factorial's argument icon. As usual since it is now fully instanti- 
ated, it executes. But now instead of being empty, there is quite a bit of code for 
factorial to execute: 
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• It cleans off the screen and repositions the factorial icon. 

• It creates a conditional icon. 

• It creates an equality testing icon, fills it in, and evaluates it Since 5 * 1, it 
evaluates to false again. 

• This causes the false branch of the conditional to be executed again. 

• This creates multiplication and subtraction icons, fills in the subtraction 
icon; and evaluates it, producing 4. 

• It creates yet another instance of the factorial icon, fills in the argument with 
4, and evaluates it. 
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And so on recursively, until finally factorial is called with 1 as the argument. At 
this point, for the first time, the conditional's true icon is executed. But there is 
no code yet for this icon, so it immediately traps back to me asking for instruc- 
tions. The screen now looks like this: 



The screen just before factorial(4) is 
evaluated 
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In this case, we know exactly what to do. Factorial(l) = 1. So I type 1 into the 
value icon: 



0 = 



and invoke the "done" menu item. This returns from the last recursive call— fac- 
torial(l). It immediately traps asking for more instructions for the false branch of 
the conditional, since it was still trying to compute factorial(2) when the recur- 
sive call to factorial(l) was made. The screen now looks like this: 
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All that remains is to drag die value from the recursive call frctorial(l) into the 
multiplication icon: 




which causes it to evaluate: 




This then is n*f actorial (n-1), where n ■ 2. 1 drag this value into factori- 
al's value icon— the false branch as well as the true branch must compute a 
value — and invoke "done" 



0 = 



The recursion now completely unwinds, since it knows how to do everything it 
needs to, and ends with the value of factorial(6). See the next page. Finally, in 
the picture on the following page, it restores the screen to its state when the 
function was initially invoked. 
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People sometimes ask me. if I were to build Pygmalion today, how would it be 
different? There are three main things I would do differently: 

1. The most important change I'd make is to address a broader class of users. 
Pygmalion was designed for a specific target audience: computer scientists. 
These people already know how to program, and they understand programming 
concepts such as variables and iteration. I would like to see if the approach could 
be applied to a wider audience: business people, homemakers. teachers, children, 
the average "man on the street" This requires using objects and actions in the 
conceptual space of these users: instead of variables and arrays, use documents 
and folders, or cars and tracks, or dolls and doll houses, or food and spices. Or 
better yet, let users define their own objects and actions. Dan Halbert captured 
this idea nicely with his concept of "progiaiiuning in the language of the user in- 
terface" [Halbert 84]. 

2. The biggest weakness of Pygmalion, and of all the programming by demon- 
stration systems that have foUowed it to date, is that it is a toy system. Only 
simple algorithms could be programmed. Because of memory limitations I 
could execute factorial(3) but factorial^) ran out of memory. (Pygmalion was 
implemented on a 64 K byte computer with no virtual memory.) There was no 
way to write a compiler in it or a chess playing program or an accounting pro- 
gram, nor is it clear that its approach would even work for such large tasks (I 
did. however, implement pan of an operating system in it, at least on paper.) The 
biggest challenge for programming by demonstration efforts is to build a practi- 
cal system in which nontri vial programs can be written. 

3. 1 would put a greater emphasis on the user interface. Given what we know 
about graphical user interfaces today, it wouldn't be hard to improve the inter- 
face dramatically. I would put the icons into a floating palette as in HyperCard, 
replacing the long menu area that takes up so much screen space. I would put the 
operauons on icons into pull-down menus. I would get rid of the bottom four ar- 
eas in the Pygmalion window entirely. The "mouse" area was necessary because 
the system was pretty modal; I would redesign it to be modeless. I would make 
greater use of overlapping windows (which hadn't been invented when this work 
was done). I would improve the graphics esthetics. With today's graphics re- 
sources-bit mapped screens, processor power, graphics editors, experienced bit 
map graphics artists-there is no longer any excuse for poor appearance Good 
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esthetics are an important factor in the users 1 enjoyment of a system, and enjoy- 
ment is crucial to creativity. 

ii^l^ir y^^^^^^^ ^ Pygmalion was an early attempt to improve the process of programming. By 

studying how people think and communicate, I attempted to build a program- 
ming environment that facilitated communication and stimulated people's ability 
to think creatively. While it never went beyond a toy system, Pygmalion embod- 
ied some ideas that still seem to have promise today: 

• It allowed ideas to be worked out via sketches on the screen and then was 
able to reexecute the sketches on new data. 

• It introduced icons as the basic entity for representing and controlling pro- 
grams. 

• Programming was done by editing sketches and then recording the editing 
actions. This avoided the abstract step of writing down statements in a pro- 
gramming language. 

• Example data were always concrete, never abstract 

• It used analogical representations for data. This reduced the translation dis- 
tance between mental models and computer models of data. 

• It represented programs as movies. (Storyboards would probably be better, & 
la David Kurlander [Kuriander 88b].) 

To make it possible for the average person to write computer programs, I still 
believe that some combination of these ideas must be present. This is a worthy 
challenge for the 1990' s. 
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Pygmalion 

Application domain: Custom graphical programming environment 
Intended users: Programmers 

How does the user create, execute and modify programs? 

The user creates programs by editing graphical snapshots of the computation. 
Essentially the user treats the display screen as an "electronic blackboard", using 
it to work out algorithms on specific concrete examples. 
Pygmalion provides graphical representations for the standard arithmetic, rela- 
tional, and Boolean operators. 

Partially specified programs could be executed. The system asks the user what to 
do next when it reaches the end of a branch. 

Programs can be modified, but then the rest of the modified branch must be re- 
demonstrated 

Inf erencing: No inferencing 

Types of examples: Multiple examples to demonstrate branches of a condi- 
tional, and recursion. 

Program constructs: Variables, loops, conditionals, recursion 
Machine, language, size, date: Smalltalk, 1975. 
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One way of giving casual users access to some of the power of a computer with- 




out having to learn formal programming methods is to enable tasks to be defined 
by giving examples of their execution, rather than by a procedural specification. 
This chapter summarizes a 1981 proposal for an electronic calculator that allows 
iterative computations to be inferred interactively from an initial part of the se- 
quence of key-presses [Witten 8 1]. The aim was to construct a device that is par- 
tially self-programming. The predictive calculator was implemented on a PDP- 
1 1/40 computer in 1981. It was intended as a demonstration rather than a practi- 
cal tool for casual users, and the user interface was not completely polished. 

The idea of defining problems operationally through examples was proposed in 
the "query-by-example" system for database retrieval [Zloof 77b], which was 
intended for a user with no programming and little mathematical experience. To 
specify what to retrieve, you present an example of an item that should be in- 
cluded. The scheme illustrates that useful interactive systems can be built that 
allow users to define non-trivial tasks by example. One of its advantages is that 
it frees users from thinking of the retrieval problem in an artificially sequential 
form: instead they can specify the links, conditions, and constraints in any order. 



Figure 1. Trace of a sorting process 



t:=a(0) s:=0 i< t -l a(i + l)<a<i) s:=i X:=a(i) 

a(i):=a(i*l) a(Ul) !stX i< t -l a(iU)>a(i) i=i*i 

i<t-l a(i + l)<a(i) s:=i x:=a(i) a(i):=a(i+l) a(i + lj:=x 
i:=Ul i<t-l a(i + l)<a{i) s:=i x:=a(i) a(i):=a(i+i) 

a(l*l):=x i:=i + l i =t -l s>0 s:=0 i:=l i< t -l a (i*l)>a<i) 

i<t-l a(Ul)>a(i) i<t -i a(i + l,<a(i, s:*i 

x:=a(i) a(i) : =a(i+l) a(i + l):« x i :=i+1 i<t . x 
a(i+l)>a(i) i:=Ui i =t -i s>0 s:=0 i :=1 ^t.j 
a(i + l)>a(i) i< t -i a(i*l)<a(i) s:=i x:=a(i) 

a(i):=a(i*l) a(Ul):= x i <t i a{i*l)>a{i) 

i<t-l a(i*l)>a(i) i: = i + l i =t -l a >0 s:=0 i =l 

i<t-l a(i<.l)>a(i) i:=i.i 1<t -i a(i + i)>a(i) i:=i + l 

i<t-l a(i + i)>a(i) i : =i + i i< t -i a(i*l)>a(i) i:=i*i 
i»C-l s=0 end 



Input: a (0). 
a(l) 



number of elements to be sorted, minus 1 
a(t-i) elements to be sorted into increasing numerical order 



In contrast, the present work explores how strictly sequential information can be 
inferred by a machine from an initial subsequence. 

Automatic inference of programs from traces is another domain in which the 
problem is specified by an example. When a detailed trace of a particular execu- 
tion of a program is available at the right level of generality, program inference 
can be easy. For instance, [Biermann 72] discusses the inference of luring ma- 
chines from traces of sample computations. The problem is more usefully ad- 
dressed at a higher level than Turing-machine instructions, and [Gaines 76] gave 
tbe programming-language-level trace in Figure 1 as an example of a sorting 
program that performs a simple bubble-sort. The flowchart in Figure 2 can be 
derived from the trace simply by assigning different states to different state- 
ments and drawing sequential links between them. It is correct and complete ex- 
cept for a link from tbe assignment "i = =1 " to the test "i=t-i" which was not 
demonstrated by that particular example. However, it is unlikely that such a 
trace would be entered without error, and even if it were, it may not be any eas- 
ier for a casual user to generate than a structural description-a program-for 
tbe sorting process. Traces can be useful, though, in more limited domains-as 
this chapter will show. 
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Figure 2. Program formed from the 
trace 



When using interactive computers, there are many situations in which it is hard 
to decide whether to do a minor, but repetitive, task by hand or write a program 
to accomplish it Simple, repetitive arithmetic calculations often present this 
quandary. For example, to plot y = jre*** for a dozen or so values of x, should 
one use a hand calculator or write a BASIC program? The first is easier and 
more certain; it will not take more than 10 minutes. The second may be quicker, 
but could require consulting the manual to refresh one's memory of the vagaries 
of BASIC syntax. 

The Predictive Calculator is a system that quietly "looks over the shoulder" of 
the user, fanning a model of the action sequence. After a while it may be able to 
predict what keys will be pressed next Any that cannot be predicted correspond 
to "input" Thus the system can eventually behave exactly as though it had been 
explicitly programmed for the task, waiting for the user to enter a number and 
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simulating the appropriate sequence of key-presses to calculate the answer. The 
user must pay a price for the service, for the system cannot help but be wrong 
occasionally; moreover, extra keys must be provided to enable predictions to be 
accepted or rejected. 

Sample dialogues 

The device is based on the simple, non-programmable pocket calculator shown 
at the beginning of this chapter. An adaptive model is constructed of the se- 
quence of keys that are pressed. If the task is repetitive (like computing a simple 
function for various argument values), the system will soon catch on to the se- 
quence and begin to activate the keys itself. When the prediction is wrong an 
undo key was envisaged to enable the user to correct it by undoing a single step 
of the sequence (though, as noted above, the interface was never completely fin- 
ished). In order to keep control in the hands of the user, predictions are indicated 
but not actually made until they have been confirmed by an accept key. One can 
imagine being able to switch into a "free-run" mode after having gained enough 
confidence in the predictions, but this was not implemented. 

Figures 3-5 give examples of this "self-programming" calculator. The first 
shows the evaluation ofxe 1 "* for a range of values of x. The keys pressed by the 
operator are in normal type; those predicted by the system are shaded. From 
halfway through the second iteration onwards, the device behaves as though it 
had been explicitly programmed for the job (except for the need to repeatedly 
press accept or switch into free-run mode). It waits for the user to enter a num- 
ber, and displays the answer. It takes slightly longer for the constant 1 to be 
predicted than the preceding operators because the system requires an additional 
coiifirmation before venturing to predict a number (see below). Provided the 
user enters variable data before any fixed constants (and people generally do 
this), there is no difficulty in getting the calculator to pause when the answer has 
been reached: it stops automatically whenever it cannot predict the next element 
in a sequence and this occurs at the end of each of the four calculations in 
Figure 3. 



Figure 4 shows the evaluation of 



1 + 



log* 



8 log 2 

for various values of x. The first line stores the constant log 2 in memory. 
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Figure J. Using the self-programming 
calculator to evaluate xs\~ x for vari- 
ous values of x 
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Figure 4. Evaluating 

1+ log x / 81og2/or various values ofx 
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Figure 5. Evaluating a more complex 
expression 
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More complicated is the evaluation of 

20 + lOlog [l + 02-2*008 gfe] - 10 log [1 ♦ ^-a.coHfl 

for a=0.9, shown in Figure 5. Since the calculator possesses only one memory 
location, u is expedient to compute me last subexpression first and jot down me 
result Hie result of this calculation (-2.69858) only has to be keyed twice be- 
fore the system picks it up as predictable. Some interference occurs between this 
mitral task and the main repeated calculation, and three suggestions had to be 
Undone by the user. 

If an erroneous suggestion is made, indicated by an "undo." me system is cau- 
cus about making a prediction in that context in the future. Any step that is un- 
done is erased from the system's model of the interaction, leaving no record of 
its existence. Of course, the user will (presumably) have followed the undo by 
an action that is different from the one predicted, creating a conflict in the 
model. Consequently when the same context occurs again, no prediction will be 
offered to the user. However, the system will internally monitor its own predic- 
tions in that context, and when several correct predictions have been made sub- 
sequendy (but not presented to the user), it will eventually venture its sugges- 
tions. This is the reason why the penultimate V on each line of Figure 5 has to 
be keyed by the user several times. Both of the sequences 

log x io = 
log X 10 + 

had occurred in the interaction, but a long run of the second was enough to cause 
it to be used eventually. 

As Figure 5 implies, any number that appears in the input sequence is identified 
and treated as a single unit To enable the predictions that follow numbers to be 
made in time for them to be useful, it was necessary to provide a special key 
with which each number is terminated. Of course, operations like "cos" al- 
though written as three letters in the figure, already correspond to a single 
keystroke and so no terminator is needed. 

The modeling technique 

The system uses an extremely simple method to form its predictions. The user's 
input sequence is characterized by the set of overlapping k-tuples of symbols 
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that occur in it, for some limited context length k (k=4 in the examples of 
Figures 3-5). The symbols are either numbers or operation keys. For instance. 
Figure 6 shows the 4-tuples that are stored for the sequence of Figure 3. 

The first row shows the first four symbols in the sequence, the second shows the 
4-tuple starting at the second symbol, the third starts at the third symbol, and so 
on. To use these for prediction, whenever the last k-l symbols entered match 
those that begin a stored tuple, the last symbol of that tuple is predicted. For in- 
stance, if the user enters "m+= +/- +" this matches the third row, so the system 
will predict 1. 

This modeling technique, known as a "length-k" modeling [Witten 79], arose 
out of context modeling techniques [Andreae 77]. It turns out that the k- tuples 
can be stored economically by massaging them into the form of an automaton 
model, although this is hardly necessary with modern store sizes. A more power- 
ful, though slightly less space-efficient, prediction technique is to use partial 
matching on the k-tuples in conjunction with a trie-structured model (e.g. the 
REACTIVE KEYBOARD [Darragh 92] and the PPM text compression method 
[Bell 90]); that would be the natural way to implement the predictive calculator 
nowadays. What is surprising is that the simplistic modeling technique that was 
actually implemented performed so well in practice. 

A modification to the modeling technique was made to suit the calculator situa- 
tion. It is clear from a cursory analysis of calculator sequences that numbers and 
operators should be treated differently, for a typical sequence comprises differ- 
ent numbers embedded in a fixed template of operators. This rule is not univer- 
sal, because fixed constants appear in the stream as well as variable input data. 
Note how the constants 1 in Figure 3, 8 and 1 in Figure 4, and 4000, 180, 2, .9, 
1, 10, 20 and 2.69858 in Figure 5 are all quickly picked out as predictable by the 
system. 

In order to prevent differences in data values from rendering the length-k se- 
quences inoperative, two parallel sets of k-tuples were used. One was exactly as 
shown above, whereas the other mapped all input numbers into the same token 
<NUM>, yielding Figure 7 from Figure 6. 

Predictions from this model were only used when the other one failed to yield a 
prediction. For example, when the first prediction of Figure 3 is made, the % I - M 
on the second line, it is predicted from the tuple "<num> mc m*= +/-", because 
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Figure 6. The basic model that is stored 
for the sequence of Figure 3 
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Figure 7. The mapped model that is 
constructed from Figure 6 



the actual context -.2 mc has not been seen before. Furthermore, the sys- 
tem was constructed to be more conservative about predicting numbers than 
operators. No prediction was made unless it would have been correct the previ- 
ous n times it occurred, and n was set differently for operators („=1) and num- 
bers (n=2). Thus the tuple «m + = + 1" is not used to predict the 1 on the 
second lme of Figure 3, but it is used on the third line. 

Except for its ability to parse numbers in the symbol string and to distinguish 
numbers from operators, the system incorporates no knowledge about the caloi- 
latoror calculation sequences. For example, it will learn nonsense sequences 
M» l l ♦ ♦ l l ♦ ♦» and regurgitate them just as readily as syntactically 
meaningful sequences. (The sequence "xx" that occurs in figure 5 may seem 
anomalous; in fact it is ore calculator's way of squaring a number.) Tne proce- 
durejs entirely lexical and does not recognize patterns of numbers. For exanq* 
■ the sequence 1. .2. .3. ... of figure 3 it is evident what should come next bui 
the system does not spot the pattern. Also, it does not notice multiple occur- 
fences of the same input data. For example, if x expfl-*) were to be evaluated 
on calculator without a memory, x would have to be entered twice. The model, 
ing procedure is incapable of exploiting this redundancy. 

A common source of variation in calculator sequences is the discovery of an 
eas.er way to do the test For example, halfway through figure 5 one may de- 
ade to enter 1.81 directly instead of -.9 xx = + x -. This is unlikely I the 
present example, for no penalty is associated with the latter sequence once it has 
been learned. However, if the simplifying discovery is matte early on in the in- 
teraction it may cause some incorrect predictions. 



Tms system ulustrates how systematic processing of a "history list" of previous 
mterattons can be used to predict future entries. Rather man being invoked ex- 
plicitly, the history list is accessed implicitly to provide assistance to the user. 
Despite the fact mat the modeler has no knowledge of the syntax or semantics of 
the dtalogue, ,t is remarkably successful. For example, when the three short 
repeuuve problems of figure 3-5 were concatenated into a single input se- 
quence, over 75% of the sequence elements were predicted and less than 1.5% 
of predtcuons were incorrect There is always a temptation to adjust system pa- 
rameters in retrospect to yield good performance, and when this was do^e 
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nearly 80% of correct predictions were achieved with an error rate of under 
0.5%. 

The basic idea behind the predictive calculator has been engineered into a device 
called the REACTIVE KEYBOARD that accelerates typewritten communication 
with a computer system by predicting what the user is going to type next 
[Darragh 92]. Obviously, as with the calculator, predictions are not always cor- 
rect, but they are correct often enough to form the basis of a useful communica- 
tion device. Since predictions are created adaptively, based on what the user has 
already typed in this session or in previous ones, the system conforms to what- 
ever kind of text is being entered. It has proven extremely useful for people with 
certain kinds of communication disability. 

It is a great pleasure to acknowledge the influence and encouragement of Brian 
Gaines, John Cleary, and especially John Andreae. John Darragh skillfully im- 
plemented the predictive calculator program on a PDP-1 1/40 computer. 
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WITTEN 




Predictive Calculator 

Application domain: Calculator 

How does the user create, execute and modify programs? 

User creates programs simply by using the calculator. There is no special team- 
ing mode. The only user interaction, beyond doing the task, is to press Undo 
when the predictor makes a mistake. 

By rejecting a prediction, the user forces the calculator to learn a new prediction 
from that context. 

Inferenclng: 

Keeps a history of all four-token sequences (a token is a number or an operator) 
Whenever the user types three tokens that match a recorded sequence, the 
Predictive Calculator predicts the fourth. (Any fixed value can be used instead of 
four"). 

Program constructs: 

Can ask for user input for any data that varies, but cannot create a variable (e a 
it does not identify the two X s in "X 2 + 5X") 

Types end sources of information: 

No syntactic rules. (e.g. it will happily learn the sequence "1 1 ++ l l + + « if 
that is what the user enters) 

Machine, language, size, date: PDP-11/40, 1982. 
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